(C) 2017-2019 by Damir Cavar
Download: This and various other Jupyter notebooks are available from my GitHub repo.
This is a tutorial related to the discussion of feature extraction for classification in the textbook Machine Learning: The Art and Science of Algorithms that Make Sense of Data by Peter Flach.
This tutorial was developed as part of my course material for the course Machine Learning for Computational Linguistics in the Computational Linguistics Program of the Department of Linguistics at Indiana University.
To extract features from text, we will need a tokenizer. In the following code we load the NLTK word_tokenize function:
In [18]:
from nltk import word_tokenize
In the following code we declare a text variable text1 and set its content to a randomly selected text from a Wikipedia article:
In [19]:
text1 = """The desert tortoises (Gopherus agassizii and Gopherus morafkai) are two species of tortoise
native to the Mojave and Sonoran Deserts of the southwestern United States and northwestern Mexico
and the Sinaloan thornscrub of northwestern Mexico. G. agassizii is distributed in western
Arizona, southeastern California, southern Nevada, and southwestern Utah. The specific name
agassizii is in honor of Swiss-American zoologist Jean Louis Rodolphe Agassiz. Recently, on the
basis of DNA, geographic, and behavioral differences between desert tortoises east and west of
the Colorado River, it was decided that two species of desert tortoises exist: the Agassiz's
desert tortoise (Gopherus agassizii) and Morafka's desert tortoise (Gopherus morafkai). G.
morafkai occurs east of the Colorado River in Arizona, as well as in the states of Sonora and
Sinaloa, Mexico. This species may be a composite of two species.
The new species name is in honor of the late Professor David Joseph Morafka of California State
University, Dominguez Hills, in recognition of his many contributions to the study and
conservation of Gopherus.
The desert tortoises live about 50 to 80 years; they grow slowly and generally have low
reproductive rates. They spend most of their time in burrows, rock shelters, and pallets to
regulate body temperature and reduce water loss. They are most active after seasonal rains and
are inactive during most of the year. This inactivity helps reduce water loss during hot periods,
whereas winter hibernation facilitates survival during freezing temperatures and low food
availability. Desert tortoises can tolerate water, salt, and energy imbalances on a daily basis,
which increases their lifespans."""
The following code lowers all uppercase characters in the text and tokenizes it. The tokens are stored in the tokens1 variable.
In [20]:
tokens1 = word_tokenize(text1.lower())
print(tokens1)
We might want to create a frequency profile from the tokens using Counter from the collections module.
In [21]:
from collections import Counter
We can create a frequency profile from the token-list in the following way:
In [22]:
fp = Counter(tokens1)
print(fp)
We could create a model that is using the token, the token frequency from the frequency profile above, and the length of the token:
In [23]:
model = [ (i, fp[i], len(i)) for i in fp ]
print(model)
We can now print the model in a tab-delimited format:
In [24]:
for x in model:
print( "\t".join( (str(x[1]), str(x[2]), x[0]) ) )
One way to manipulate the feature set extracted from text is to remove all stopwords, to model distributional properties of tokens based on function words (or stopwords). To use English stopwords, we use the NLTK stopword list. We can add tokens to the stopword list, as for example the missing pronoun us:
In [26]:
from nltk.corpus import stopwords
stopw = stopwords.words("english")
stopw.append("us")
print(stopw)
In the following code we declare a function isStopword that returns true, if the parameter is a stopword, and otherwise false. This way we can create a model of words that uses the frequency and the length of a token. The third column provides the class definition, that is 0 for non-stopword, and 1 for stopword.
In [27]:
def isStopword(word):
if word in stopw:
return(1)
return(0)
for x in model:
print( "\t".join( (str(x[1]), str(x[2]), str(isStopword(x[0]))) ) )
You can copy and paste the model above and use it directly in Timbl.
In the next part we will make use of part-of-speech tags for classification. In order to use the PoS-tagger in NLTK, we need to import the pos_tag module:
In [28]:
from nltk import pos_tag
We can use the token-list from the text above and PoS-tag each token. In the code below we abbreviate the tag to the initial character only, which marks up the main part-of-speech class, ignoring feature details.
In [29]:
tokens1 = word_tokenize(text1.lower())
posTokens = [ (x[0], x[1][0]) for x in pos_tag(tokens1) ]
print(posTokens)
For mapping of tags on numerical feature vectors, we might need the list of all tags used in the language data above. We apply list comprehension to posTokens, using only the tag element from the tuple above. The list of all tags we convert into a list to remove duplicates. The resulting set is converted into an ordered list.
In [30]:
tags = list( set( [ x[1] for x in posTokens ] ) )
print(tags)
To keep track of the frequencies of all tags left and right of any given token, we create dictionaries that have Counter-objects as values. The keys of the top-level dictionaries leftOfToken and rightOfToken are the token tuples that consist of token and tag. The keys of the Counter-values for the token keys are tags. The values associated with the tag-keys are frequencies. leftOfToken counts the frequency of tags to the immediate left of a token. rightOfToken counts the frequency of tags to the immediate right of a token.
In [31]:
from collections import defaultdict
leftOfToken = defaultdict(Counter)
rightOfToken = defaultdict(Counter)
lenPosTokens = len(posTokens)
for i in range(lenPosTokens):
token = posTokens[i]
if i > 0:
ltag = posTokens[i - 1][1]
leftOfToken[token][ltag] += 1
if i < lenPosTokens - 1:
rtag = posTokens[i + 1][1]
rightOfToken[token][rtag] += 1
We can verify that the frequency tables have been created correctly:
In [32]:
print(leftOfToken)
To generate a model, we create a numerical vector with the frequencies of the tags left and right of each token. The frequencies are sorted by position of the tag using the tags-list created above. The left half of the numerical vector represents the left context, the right half the frequencies of the tags to the right of a token. The final element of the row is the tag of the token that is represented by the tag-context frequency vector:
In [33]:
for token in leftOfToken.keys():
leftVector = []
rightVector = []
for tag in tags:
leftVector.append(leftOfToken[token][tag])
rightVector.append(rightOfToken[token][tag])
print(" ".join([ str(x) for x in leftVector ]), " ".join([ str(x) for x in rightVector ]), token[1])
The model above can be used as a training model with Timbl. Copy and paste it into a training model file. In the following we will create another model from some random Wikipedia text. We generate the model for this new text as above.
In [35]:
text2 = """Desert tortoises spend most of their lives in burrows, rock
shelters, and pallets to regulate body temperature and reduce water loss.
Burrows are tunnels dug into soil by desert tortoises or other animals,
rock shelters are spaces protected by rocks and/or boulders, and pallets
are depressions in the soil. The use of the various shelter types is
related to their availability and climate. The number of burrows used,
the extent of repetitive use, and the occurrence of burrow sharing are
variable. Males tend to occupy deeper burrows than females. Seasonal
trends in burrow use are influenced by desert tortoise gender and
regional variation. Desert tortoise shelter sites are often associated
with plant or rock cover. Desert tortoises often lay their eggs in
nests dug in sufficiently deep soil at the entrance of burrows or
under shrubs. Nests are typically 3 to 10 inches (8–25 cm) deep.
Shelters are important for controlling body temperature and water
regulation, as they allow desert tortoises to slow their rate of heating
in summer and provide protection from cold during the winter. The
humidity within burrows prevents dehydration. Burrows also provide
protection from predators. The availability of adequate burrow sites
influences desert tortoise densities.
The number of burrows used by desert tortoises varies spatially and
temporally, from about 5 to 25 per year. Some burrows are used repeatedly,
sometimes for several consecutive years. Desert tortoises share burrows
with various mammals, reptiles, birds, and invertebrates, such as
white-tailed antelope squirrels (Ammospermophilus leucurus), woodrats
(Neotoma), collared peccaries (Pecari tajacu), burrowing owls (Athene
cunicularia), Gambel's quail (Callipepla gambelii), rattlesnakes (Crotalus
spp.), Gila monsters (Heloderma suspectum), beetles, spiders, and
scorpions. One burrow can host up to 23 desert tortoises – such sharing
is more common for desert tortoises of opposite sexes than for desert
tortoises of the same sex.
"""
tokens2 = word_tokenize(text2.lower())
posTokensB = [ (token[0], token[1][0] ) for token in pos_tag(tokens2) ]
leftOfToken2 = defaultdict(Counter)
rightOfToken2 = defaultdict(Counter)
lenPosTokensB = len(posTokensB)
for i in range(lenPosTokensB):
token = posTokensB[i]
if i > 0:
ltag = posTokensB[i - 1][1]
leftOfToken2[token][ltag] += 1
if i < lenPosTokensB - 1:
rtag = posTokensB[i + 1][1]
rightOfToken2[token][rtag] += 1
for token in leftOfToken2.keys():
leftVector = []
rightVector = []
for tag in tags:
leftVector.append(leftOfToken2[token][tag])
rightVector.append(rightOfToken2[token][tag])
print(" ".join([ str(x) for x in leftVector ]), " ".join([ str(x) for x in rightVector ]), token[1])
You can copy and paste this model into a new test-file for Timbl.
(C) 2017-2019 by Damir Cavar - Creative Commons Attribution-ShareAlike 4.0 International License (CA BY-SA 4.0)